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first two pages were read out loud, and then students read silently 
the next six pages of the text with two interruptions at separate 
points in the story. In one class, students were given a set of 
written questions to complete at each interruption point. The other 
class was asked to complete a sketch of what they thought would 
happen next (the "Sketch to Stretch" strategy) . Results indicated 
that: (1) use of Sketch to Stretch provided a different type of data 
on which to make predictions about student understanding of text; (2) 
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involvement, interest, and concentration in the experimental lessons 
than in traditional lessons. Findings of this and the other micro 
case studies give legitimacy to a variety of "non-questxon* 
comprehension strategies as vehicles for assessment and learning. 
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BEYOND THE QUESTION: AN EVALUATION OF 
ALTERNATIVE STRATEGIES FOR THE ASSESSMENT OF 

READING COMPREHENSION 



Trevor H. Cairney 



Introduction 

My concern within this section of the symposium is with the diagnostic assessment of 
reading comprehension. I am reluctant to use the word diagnostic because of its clinical 
connotations, but it is a useful term. It is a word based on the Greek 'diagnostics', 
which broadly means to be able to make comparisons or to distinguish between specific 
phenomena. 

In this short paper I want to provide insight into field testing in which I have been 
involved using a range of process-based reading comprehension assessment devices. 
Each of these has been developed as an outcome of comprehension research (see 
Cairney, 1989; 1996a; 1990b) exploring the use of alternative modes for heightening 
student engagement with text 

My work has been a natural outgrowth of a general trend in reading research that has 
emphasised an interest in reading- writing relationships (Shanklin, 1982; Tierney and 
Pearson, 1983; Cairney, 1985; 1988), the active role of the reader (eg. Kamil, 1984; 
Rowe, 1984; Cairney & Langbein, 1989), and a desire to explore the potential of text to 
offer unlimited meanings (Bardies, 1979; Cairney, 1990a; 1990b; 1990c; Eco, 1979; 
Rosenblatt, 1978). This work has converged to create my current interest in the 
assessment of reading comprehension. 

It was Barthes who wrote that the text is: 

"completely woven with quotations, references, and echoes. These are cultural 
languages ... that traverse the text from one end to the other in a vast 
stereophony ....Every text, being itself the intertext of another text {Barthes, 
1979. p. 77)." 



It seems that much of comprehension assessment reflects an assumption that what is 
being assessed is the ability to transfer information that is in some way almost 
magically encoded within texts. I want to challenge this assumption and argue that 
reading comprehension is a highly constructive act involving the construction, 
reconstruction and transformation of meaning. I want to argue that if this complex 
process is to be assessed, it requires strategies that provide access to this process of 
meaning making, not simply tests of information transfer or reproduction. 

I used the expression "beyond the question" in my title not to devalue the use of 
questions as a tool for both teaching and assessment, but rather to signal that I have 
been exploring alternative strategies that attempt to sample a wider cross section of the 
meanings that readers are constructing as part of the reading process. 



Let me share an example to illustrate what I mean. What follows is a segment from a 
text about the history of man's attempts to conquer Mount Everest and a question that 
was used to assess student understanding of the text. 

Text segment: 

from then on Everest was a magnet that attracted mountaineers from 

many nations. They would come to it to fight against its many perils of ice. 
wind and snow 

Question: 

In the passage, Mount Everest was compared with a: 

a) blizzard 

b) magnet 

c) battle 



Now the required answer was clearly "magnet". But please note what this assessment 
item does, and does not do. What it does do is test the ability of students to identify a 
metaphor within a text, but it does little to provide knowledge about the range of 
responses possible, and the students' reasons for offering them. Why might a student 
respond by circling a) for "blizzard"? Could a child's selection of c) reflect his/her 
attempts to construct meaning? This item shows us the students who can and cannot 
identify this metaphor, in this text, but it does little to provide knowledge about our 
students as meaning makers. 

We need to continue to challenge teachers to extonine how questions are used. If used 
to encourage readers to share meanings and insights they are a powerful tool for 
learning and assessment But if they act simply as tools to lead readers to a single 
meaning that is assumed to be in the text, then they are very limited. This restricted use 
of questions is designed to narrow a student's focus to a single meaning. The former is 
designed to build meaning, to open up the "endless possibilities" of text (Cairney, 
1990a). One use of questions seeks simply to (est whether a unitary meaning has been 
comprehended, the other is designed to help students construct their own meanings. 
Student answers to the latter provide us with insights into the nature of the 
comprehension process that the more restricted use of questions does not. 

A further problem with the strategies (alludec to above) that have been used previously 
to assess reading comprehension, is that the? fail to recognize the critical relationship 
that should exist between teaching and assessment. Teaching and assessment are 
frequently viewed as separate and largely unrelated activities. As a result, there is often 
a gap between the type of strategies used for teaching and assessment. 

The purpose of this small study was to explore alternative strategies for assessment, 
and to explore what happens when assessment is used as an integral part of teaching 
and learning. 

Background to this paper 

The focus of this paper is perhaps not what most would expect. It is not simply a 
description of the strategies used Nor is it a study which correlates data generated by 
my strategies with those generated with traditional assessment strategies. Both of these 
approaches are quite legitimate. What I want to do in contrast is provide a detailed 
description of how these alternative strategies can be used, and what type of 
information they offer. 
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I want to do this in the short time available by describing the use of a specific text 
within an authentic classroom context. In the process, I want to try to provide an 
insight into the complex interrelationship of teaching, assessment and learning. 

In the time I have available I intend to discuss one of a series of micro case studies of 
specific assessment strategies that I conducted in 1989 within an Australian elementary 
school. The decision to describe just one case study is motivated by a desire to provide 
a detailed qualitative analysis of one slice of data, rather than a superficial treatment of 
multiple slices of data. 

Each of these micro case studies followed an identical pattern. Two similar ungraded 
classes were chosen in the same school, and a text shared with each set of students in 
the same way. The students were asked to read the text (sometimes orally, sometimes 
silently, and occasionally using a combination of both), and were assessed during the 
reading to gain insights into their comprehension. One group always experienced 
'traditional' question and answer strategies, while the other experienced a variety of 
alternative 'non-question' based strategies. 



Field Testing of Sketch to Stretch 

The example I want to share involved the mailing of a strategy known as Sketch to 
Stretch (described above). This strategy was trialled in two parallel grade 5 classes ( 10 
year olds) in the same school. Each class was introduced to the picture book The 
Wedding Ghost (Garfield, 1985) in exactly the same way (see Table 1). The text was 
introduced by showing its cover, discussing the tide, providing a quick biography of 
the author, and explaining that it was an example of historical fiction. The first two 
pages of the text were read out loud After this, students read silently the next 6 pages 
of the text with two interruptions at separate points in the story. 

In one class the students were given a set of written questions to complete at each of 
these interruption points. For convenience I will call this the control group. Readers in 
this group were invited each rime to respond to the questions by providing written 
answers. They were told prior to answering them that many interpretations were 
possible, and that there was not necessarily a single correct answer for each question. 
The other class was asked to complete a sketch each time. Once again, for 
convenience, this class will be called the 'experimental' group. 

Responses from the 'control' group to the first set of questions (see Table 2) were 
generally poor. This was surprising because they were relatively simple. Many 
students were unable to offer suggestions for specific questions. In fact 37.5% of all 
questions were unanswered. 
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Table 1: Detailed procedures for the text The Wedding 

Ghost (Garfield) 



GROUP 



PROCEDURES 



Experimental * Read the first pan of The Wedding Ghost up to P. 5 (Slop after - 

' her topay her fare and bring a gift besides.") 

* Ask the group to sketch what they think will happen next. 

* Read on till P. 9 (Stop after - "He shivered and tore open the parcel.") 

* Ask the group to sketch what the contents of the parcel might be. 

* Continue story up to P. 11 (Stop after - "he turned up his collar and ran 
like a thief ."). 

* Ask the class to answer a scries of questions about the story and predict 
the ending. 

Traditional • Read the first part of The Wedding Ghost up to P. 5 (Stop after - 

' her to pay her fare and bring a gift besides.") 

* Provide written questions to be answered about the story so far. 

* Read on till P. 9 (Stop after - "He shivered and tore open the parcel.") 

* Provide more written questions to be answered about the story so far. 

* Continue story up to P. 1 1 (Stop after • "he turned up his collar and ran 
like a thief."). 

* Ask the class to answer a series of questions about the story and predict 
the ending. 



Table 2; Questions used with the 'traditional' assessment group during 

the reading of The Wedding Ghost. 



First set of questions 

1 . What is the name of Jack's uncle? 

2. What are Jack and his family preparing for? 

3. Who was the one person the Goodmans had not invited to the wedding? 

4 . Why might the Goodmans have left this person off the invitation list? 



1. 
2. 
3. 
4. 
5. 
6. 



Second set of questions 

What were some of the presents that Jillian and Jack had received? 

Who were some of the other people watching the opening of the presents? 

Why did Jillian get upset with Jack? 

What was unusual about the last present to be opened? 

How did Jack feel about Uncle Goodman's joking? 

What do you think was in the present? 
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The level of success on different questions varied considerably. For example: 



• * 50% could not name Jack's uncle; 

• * 19% were unable to indicate for what Jack and his family were 
preparing; 

• * 31.25% could not suggest who the person was who had not 
been invited to the wedding; 

• * and 50% were unable to suggest why this person was left off the 
invitation list (something not stated in the text). 



Responses to the second set of questions (see Table 2) were once again poor for the 
control group. Overall students failed to offer answers for 29. 1% of all questions with 
a range from 19% (question 6) to 56.25% (question 3) for specific questions. 



When the quality of responses was examined it became even more difficult to make 
judgements about student understanding of the story. While many could offer pieces of 
information from the text (e.g. over 50% could list some presents received, ana name 
the major characters), few seemed to be aware of the subtleties of the relationships 'hat 
Garfield was attempting to develop, nor the intricacies of the plot. The latter is probably 
indicated by the failure of most studentf to offer a satisfactory answer for questions 3 to 
6. However, this is difficult to determine given the paucity of data generated by the 
questions. 



In contrast to the control group, there was no difficulty obtaining responses from the 
experimental group. This is not surprising since the tasks were open, students being 
required simply to use Sketch to Stretch activities at each point when the reading was 
interrupted. After the first interruption students were asked to draw a sketch depicting 
what they thought would happen next. The responses to this suggestion were diverse 
in detail (see Figure 1), but close analysis of the sketches suggested that there were a 
number of distinct sub-categories. In all, 75% of students drew sketches indicating a 
wedding would occur, while the remaining 25% indicated that the ghost of the maid 
would appear in some form. 
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Figure 1: Sketch to Stretch responses after the first interruption for The 

Wedding Ghost. 
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The second sketch occurred just before Jack opened the mysterious present. Students 
were asked to predict what might be in the present and to sketch it. Once again the 
responses were diverse (see Figure 2) but tended to fall into a number of distinct 
categories. The most common responses were to draw a ghost (43%), or some type of 
letter or instructions (19%). A variety of other objects made up the remainder of the 
responses. These included, a key (2 students), crucifix, clock, book, key, ring and 
picture. 



Figure 2: Sketch to Stretch responses after the second interruption for 

the The Wedding Ghost. 
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As well as the questions and sketches used during the reading of the text, an identical 
post-reading task was provided for each group. It consisted of a series of three open 
ended questions. Students were told that there were no single correct answers for these 
questions, and that they were being asked for their opinions about the story. Students 
were asked to look at the questions (see Table 3). consider their ideas, and write the 
answers on a sheet provided. The purpose of this final task was to provide 
comparative data concerning the impact of the alternative assessment tasks on 
comprehension. 

Table 3: Post-reading task questions for The Wedding Ghost 



Questions 

* Why did he hide the map in his pocket and take it with him? 

* Why did he feel like a thief? 

* How will this story end? 



Responses to these questions varied between the groups. The first form of analysis 
was simply to look at the fluency or length of the response by conducting a word count 
for all students. This revealed for example, that the control group had an average 
output of 10.0 words for Q.l, 7.5 words for Q.2 and 18.2 words for Q. 3. In contrast 
the experimental method group had average outputs of 9.0 words for Q.l, 11.1% for 
Q.2 and 28.6% for Q.3 (see Table 4). 



Table 4: Mean word counts for responses to question 1 of the post- 
reading task. 



Mean Wond Count 



Question Traditional Group Experimental Group 



1 10.0 9.0 

2 7.5 11.1 

3 18.2 28.6 
Ave. all Questions 1 1 .9 16.2 



While there is little difference between the groups for question 1 there are significant 
differences for questions 2, 3 and the average word rate for all questions. While one 
would not want to place too much importance on word output alone, it seems plausible 
that this finding reflects the fact that the group that used Sketch to Stretch had been 
more engaged in the reading, and hence constructed more elaborate mental texts. 

Qualitative analysis of the responses supports this conclusion. The purpose of 
Question 1 was to determine if students were aware of the underlying intrigue that was 
being developed in the text. Were the students aware that the map would have a great 
deal of significance later in the story? Had they predicted that it could be linked vitally 
with events to occur later? Close analysis of the responses indicates that there were 
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differences between the two groups. Responses of control students showed little 
evidence that predictions of this type were being made. The most common response 
(35.3%) suggested Jack took it "because he wanted to work it out" or "to see what the 
place was". Other responses included "because he wanted to do this" (2 students), "to 
find the treasure" (2 students), "he was able to re*d it", and "to get to the river". Only 
two students saw anything threatening or sinister about the map - "he was afraid of it", 
"he had an urge in his body". 

In contrast, a larger number of the students in the experimental group produced 
responses suggesting that they saw deeper underlying implications for his removal of 
the map (38.1%). While a number of students once again said he took it because he 
wanted to follow the map or determine what it was for (33.3%), or simply that "he 
didn't know why" (28.6%), the most frequent response indicated that high level 
predictions were being made. These students saw deeper reasons for Jack's removal of 
the map. For example: 

"Because it was as though it told him to and he wanted to find 
where the map directed" 

"He does not know it could be some mysterious force that pushes 
Jack and the map together" 

"Because he didn't want anyone to read it" 

"He feels its got something to do with his marriage" 

"Because he was out of his mind" 

"He didn't know what it was and was trying not to show anyone". 



Responses to question 2 were far less helpful and most students simply suggested that 
Jack felt like a thief because "he took it". The failure to observe qualitative differences 
between the groups probably reflects the quality of the question. On reflection, the 
question was far too obscure for these year 5 students. It was unable to achieve its 
purpose of uncovering whether students were aware of Jack's growing fear and 
uncertainty about the events that had taken place. 



Responses to question 3 provided by far the most striking indication of differences 
between the groups. This question had been included to enable students to predict the 
ending, and in doing so, to provide a window into the texts they had constructed as part 
of the reading. Were they aware that this story was likely to move in unexpected 
directions? Had they predicted the map may be linked closely with the uninvited guest? 
Was there evidence that they were aware of a number of underlying themes which 
appeared to be emerging in the story (eg. jealousy)? 



The responses suggest that students in the traditional method group were largely 
unaware of these things, whereas the experimental group was aware that something 
mysterious was going to happen. Responses from the traditional group fell into two 
major categories - "they lived happily ever after" (35.3%), or, "he found the treasure" 
(47.0%). One student indicated that "the Wedding Ghost would get him", but the 
remaining responses were all quite specific and in some cases unrelated to the text - "the 
person following him will get caught", "he will start crying like a baby", "Jack will kill 
himself. 




In contrast, the responses from experimental group students fell into two quite different 
categories. By far the largest number of students (42.9%) indicated that a ghost would 
appear in some form (the nurse) and would either come between Jack and Jillian or 
simply ruin the marriage, stopping it from going ahead. Another group of students 
(23.8%) felt that either Jack or Jillian would die. Only two students felt that Jack and 
Jillian would be married. The remaining individual responses included - "... he will 
turn into Jack the Ripper "... he will find some treasure "... they will have a 
holiday". 

It appears that the experimental students were aware that the story had a number of 
deeper layers of meaning, and that something mysterious or sinister was about to 
happen. This suggests that this group had constructed qualitatively different texts as 
part of the reading. 



Discussion 

While only a small slice of the data from my research study has been reported here I 
believe a number of conr'usions can be drawn. First, the use of strategies like Sketch 
to Stretch provide a different type of data on wnich to make predictions about student 
understanding of texts. While it could be argued that the use of questions could have 
been made more elaborate in order to tap some of the process data, the drawings 
nevertheless, provided different data. 

Second, the evidence within this study suggests that alternative non-question 
techniques have the potential to simultaneously fulfil the dual tasks of assessment and 
teaching. Strategies like Sketch to ^:retch have great potential for stimulating student 
engagement with texts. While this is not to deny that questioning can also fulfil these 
dual roles, the use of alternative strategies appear to provide different windows on the 
reading process. 

Third, the data also suggest that the use of alternative strategies like drawing, may have 
the potential to help readers construct qualitatively different mental texts. 

Fourth, the varied outcomes on the post-reading task may suggest that an additional 
factor influencing the different impact of the experimental strategies compared to 
questioning, is that the former provides greater scope for the sharing of responses 
between readers. Strategies like Sketch to Stretch, oral discussion, character rating 
scales etc, gain their strength from the fact that they invite and encou* ~>ge readers to 
share responses and insights about reading. In doing this they implicitly 'invite' 
readers to read on to make new discoveries. In contrast, one suspects th&t students 
who are subjected to the traditional written question format are simply being 
encouraged to read on so another task can be completed This suggests mat for some 
readers, deep engagement with text requires more support. 

Fifth, when students experienced the experimental lessons they displayed greater 
involvement, interest and concentration. Students enjoyed the lessons and often 
wanted the teacher to continue after the allotted time. In contrast, within the traditional 
lessons the same students displayed more limited attention spans, appeared to be less 
involved in the reading and lesson activities, and were generally glad when the lesson 
was over. 

Implications for classroom practice 

A number of alternative assessment strategies have already been field tested as part of 
this study. 
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These include the following: 

a) Story mapping 

Story mapping is a simple strategy that requires students to draw a pictorial 
representation of a setting for a narrative text (Cairney, 1990a). Students are 
asked to imagine the setting (or part of the setting) for a particular story and to 
produce a represr .nation of it as it would appear from the air (see Figure 3). 

b) Sketch to stretch 

This strategy also uses drawing but is linked closely with text prediction and 
imagery (Cairney, 1990a). At specific points within a narrative text students are 
asked to produce a sketch which represents what they think will happen next, 
what has just taken place, how they visualize a particular event or character etc 
(see Figure 3). 

Figure 3 : Sample assessment strategies used as part of the project 

(Cairney, 1990a) 
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Sketch to Stretch for 
"Fair** Fair" (Garneid) 
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Rating Scales for "The Shrinking 
of Treenorn" (Heide) 
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c) Character rating scales 

Character rating scales require students to make judgements about characters in 
accordance with a series of personality traits (Cairney, 1990a), for example, 
intelligence, kindness etc. Students are either given the traits or are asked to 
suggest them. They then provide a rating of specific characters using a sliding 
scale that represents each personality trait (see Figure 3)* 

d) Talk to the Author 

I first developed this strategy purely for teaching (Cairney, 1986; 1990c), but 
recently have been exploring its use as an assessment device. Essentially it 
involves a written dialogue between a leader and the implied author (see Figure 
4). Readers are encouraged to ask questions of the author, disagree with 
him/her, seek clarification, offer opinions, add extra information. This strategy 
can be used with either narrative or factual texts. 



Figure 4 : Example of the Talk to the Author Strategy (Cairney, 1990b) 



Ob ho reteagg, he stayed «* of troutfe ft* 
^ymoreihan iwrt&*&km$ ************* 





wen* prospecting (or gpld Whcnthf sqyauentn 
the ansa reported hones and twite masmj. iKe 
potar suspected Ned Hww, they could not 



afted in the press. <n> the befmrtmg of the 
"tUUy Otrt&fca*" Newspaper* m the atfnflr of 
Vjciooi iwr full of report* of the KcUy gwg 
The* bad been declared ouil**% b> the 
government, »h*h meant thai anyone covKS 
shoot them on sighi ' 




fai Apni HIS. seven months after the BeniJii 
arrest, one of the potacmcn m the fight. 
Constettfe Ftupamck. went to ihe Kelly 
homestead it Gretj 10 investigate *n>rse stealing 
The constable claimed iha m the homestead he 
%r*s chot tt by Ned KeBy, bashed or Mn KdJy 
and had his gun stolen by Ned * brother. Dan. 
whjfctwone^bws w*tctwd Thr Kettys argued 
that Rupetrtck va drunk and to bUme for the 
modem They 51*10* lhat Ned •wasn't e*en there 
and hadn't been home fr* three weeks 



The gang robbed the bank at lutoi and *n 
New South Wales took over the to*™ of 
JenWene There, the gang took (he poi*cr by 
twfphte and parked up the mi»n street 
diigwtsed >f> police 41 m forms Ned wanted 10 fV>^ 
trie locaJ newspaper editor Ned bad wntten a 
tetier defending himself and wanted a panted 
But the editor had earhct wnuen a story about 
"the control* murders at Smngytark Creek" 
He was afruo that Keiir mrght have \ctt\ the 
story, so be tell hts emre and hid in a creek until 
the gang left town 



8*0 WITHDRAWAI 



A tool scnoofeeachcr w» go*ng into ihe 
icnWerw bank c the gang was wifktng out. 
loaded with the bank't cash Ned asked him 
whathewioied Theteadter repJted ihat he had 
come 10 maae a mthdrewi) 



"You're too late Ned Kefty has withdrawn ft 
aU.*" laughed Ned 



MAS KELLY JUWICSTED 




Authooixs tn New South Wjjcs md Vtciona 
were ooiraged by the KriSy gang A reward rrf 



Mrs KeMy and the t»o nrtghhof* were 
arresied Warwms were of f«»t "he *r*cst pi 
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A number of additional strategies are still to be mailed. The next stage of the project 
will study these additional comprehension strategies, will involve longer periods in the 
classrooms, and observation of both structured and unstructured responses. 

The most important implication of this study to date is that it gives some legitimacy to a 
variety of 'non-question' comprehension strategies as vehicles for assessment and 
learning. The data suggest that rather than being simply enjoyable alternative activities, 
they have the potential to produce qualitatively different responses from students. They 
also appear to provide different diagnostic data for teachers, and lead to different levels 
of engagement with text. 

The findings also point to the need for teachers of reading to rethink the role assessment 
plays in their reading classroom. I believe that we need a new set of principles for 
assessment, which in brief are as follows: 

• * Assessment should wherever possible examine the processes 
of meaning construction involved in reading; 

• * Hie data one attempts to obtain may take any form as long as 
they provide insights into the meanings readers are constructing; 
there are multiple ways to assess comprehension; 

• * All data is interpretable and should provide insights into the 
processes employed by students; 

• * Assessment is integral to what is happening in the classroom; 

• * Assessment tasks should reflect the complexity of the act that 
is being assessed; 

• * Assessment tasks should aid reading by heightening reader 
engagement with text. 

I believe that if we are to produce mc s readers in our classrooms who actively engage 
with texts, then we need to rethink our instructional strategies, and to incorporate 
assessment as an integral pan of our teaching. Part of the answer, is to consider 
alternative models for assessment and teaching. In short, we need to find ways as 
teachers to invite readers to expand the meanings they construct as they read; and in the 
process, find ways to make predictions about the complexity of the meaning making 
that is occurring as they read. We need to integrate assessment with teaching, not 
simply provide tests to determine if students can reproduce meanings that we as 
teachers have decided are 'in the text'. 
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